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REGISTRATION OF MULTI-MODALITY DATA IN IMAGING 

The present invention relates to the registration of images of different 
modalities, in particular so that such images may be displayed together, accurately 

5 superposed upon one another. 

There are many fields in which it is useful to image a subject using different 
modalities. For instance, one modality might provide detailed structural information 
about the subject, for instance an x-ray image or a magnetic resonance image, while 
another modalities might provide information about different structures not visible in 

1 0 the first modality, or information about functions occurring within the subject, such 
as by the introduction into the subject of a radioactive marker. While such different 
modality images can be considered side-by-side by someone trying to use the 
information given by the two different modalities; it is often useful to display the 
images in superposition one upon the other. It is clearly necessary for the 

15 superposition to be accurate, in other words for areas representing a particular 
position in the subject in one image to be accurately positioned in registration with 
corresponding areas in the other image. The process of achieving this alignment is 
known as "registration". It is particularly useful because the display of the superposed 
images allows the information from the different modalities to be interpreted very 

20 easily by a user, For example, the information about function from one modality can 
be related accurately to the detailed structural information from another modality. 

A variety of techniques for registration of images of different modalities, 
particularly in the medical imaging field, have been proposed. For example, detailed 
anatomical information about the structure of the body can be obtained from 

25 traditional x-ray images. Information about the metabolic function in the body can 
be obtained from different modalities, such as nuclear medicine. In a typical 
technique a radioactive marker is fixed to a physiological tracer. The tracer is 
injected in the blood of the patient and fixes to the cells in the patient according to 
the metabolic activity (consumption of oxygen, or glucose etc). A detector is used to 

30 detect the disintegration of the radioactive marker and to provide a corresponding 
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image whose intensities correspond to the amount of radioactive marker in each 
region. The results of several scans may be combined together in the process known 
as tomography to provide 3-D information. Typical nuclear medicine techniques 
include photon coincidence detection (PET) and single photon emission 
5 computerised tomography (SPECT). Such images are typically called emission 
images. One particularly important application of them is to the detection of tumours 
in the body. Such tumours are prominent in an emission images because of the high 
metabolic activity in and around the tumour. In the accompanying drawings Figure 1 
shows an example of an SPECT image, with areas of higher metabolic activity shown 

10 in lighter colours. 

For some years such emission images have been acquired simultaneously 
with another image, called a transmission image, such as a single photon 
transmission computerised tomography (SPTCT) image, which is obtained by 
placing a source of radiation on the opposite side of the subject's body from the 

1 5 detector. This provides an image which provides information regarding the 

attenuation and scattering characteristics of the subject's body. Figure 2 shows an 
example of such a transmission image. This transmission image is used to correct 
the emission image for the different attenuation and scattering occurring in different 
areas of the body. The need for such correction is illustrated by Figures 3 and 4 of 

20 the accompanying drawings. The problem is that in an emission image, photons 
emitted from different parts of the body, e.g. from different depths or from different 
structures, will undergo different attenuation and scattering. This will lead to varying 
intensities in the image which are not related to the metabolic function in the body. 
The transmission image effectively provides information about the differing 

25 attenuation and scattering and allows correction of the emission image. Figure 3 
illustrates an emission image of a uniform cylindrical phantom before correction. It 
can be seen that although the phantom is uniform, and thus should have a uniform 
intensity, in fact the central areas are darker because of greater attenuation. Figure 4 
illustrates such an image corrected by means of a transmission image. The 

30 uniformity is restored. 
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A problem with nuclear medicine images, such as emission images, is that 
while they give good information about function, they do not give very good 
information about the structure of the subject. In particular, the exact location of 
regions of high metabolic function cannot be accurately determined. This is because 

5 the emission image does not show much structural detail. Many other imaging 
modalities reveal detailed structure, but obviously not the functional information of 
the nuclear medicine images. However, the lack of common information, i.e. the fact 
that the nuclear medicine images do not include detailed structure, and the fact that 
the other modality images do not include functional information, means that using 

10 the information from the two images, e.g, by matching the two images in order to 
register them accurately, is difficult. Figure 5 of the accompanying drawings shows a 
typical detailed structural image obtained, in this case, by x-ray imaging. 

US Patent 5,871,013 and US Patent 5,672,877 .both disclose methods of 
registering functional nuclear medicine emission images with other modality images, 

1 5 such as x-ray images, by using the transmission image (e.g. Figure 2) as a stepping 
stone in the registration process. This takes advantage of the fact that the 
transmission image is customarily obtained on the same imaging equipment as the 
functional emission image and so is inherently registered (Le. the same pixel address 
in the image frame corresponds to the same position in the body). Furthermore, the 

20 transmission image includes some structural information, such as the edge of the 
body and the edges of the lungs, which are also visible in the different modality 
structural image. Therefore the registration of the transmission image with the 
different modality structural image is easier than direct registration of the functional 
emission image with the other modality image. In both patents, by first deriving a 

25 mathematical relationship mapping between the transmission image and the other 
modality image, a mapping between the functional emission image and the other 
modality can also be obtained because the relationship between the transmission 
image and the functional emission image is already known. Thus registration of the 
functional emission image with the other modality image is achieved without any 

30 matching process between the two images. 
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In accordance with the present invention there is provided: - 
a method of registering images of different modalities, comprising: 
taking a first image of a subject obtained by an imaging process of a first 
modality; 

5 taking a second image of the subject obtained by an imaging process of a 

second modality, said second image having a known positional relationship with the 
first image; 

taking a third image of the subject obtained by an imaging process of a third 
modality; 

10 distinguishing between at least one area of interest and at least one other area 

not of interest in the second image; 

on the basis of said known positional relationship identifying said at least one 
area of interest and other area not of interest in the first image; 

registering the first and third images by an image matching process based on 

1 5 said at least one area of interest identified in the first image. 

The at least one other area not of interest may be image of background 
outside the subject, and possibly, in medical images, areas such as the lung cavity 
within the body. Preferably the image matching process is conducted by looking 
only, or mainly, at the areas of interest. One way of achieving this is to set the 

20 intensities of the areas not of interest to a constant value, e.g. zero, so that the second 
image has been used, effectively, as a mask to mask out areas which are not of 
interest, However the area not of interest may be used to an extent in the registration 
process, and in this case the second modality image is being used to segment into the 
two areas, rather than to exclude one of them. 

25 The first and second images are preferably obtained on the same imaging 

apparatus, thus providing a known positional relationship, e.g. by being inherently 
registered. The first image may be an emission image in which intensity values are 
related to function in the subject, such as a PET or SPECT image. The second image 
may be a transmission image of the type mentioned above. The third image may be 

30 an image providing detailed structural information, such as an x-ray image, magnetic 
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resonance image or ultrasound image. 

The step of registering the first and third images may comprise deriving a 
positional transformation which maps to each other areas identified in the matching 
process as corresponding to each other. The matching process may be based on 
5 intensity or edge detection or another of the known techniques for matching two 
images. 

Particularly in the medical field, the second image may be used as explained 
above to correct the first image for attenuation, and the first image may be further 
processed as is conventional, e.g. by equalisation, 
10 The invention may be embodied in a computer system for processing data sets 

encoding the images, and the invention extends to a computer program for executing 
the method on a programmed computer. The invention also extends to a computer 
program product carrying such a computer program. 

The invention will be further described by way of example, with reference to 
15 the accompanying drawings in which:- 

Figure 1 is a functional emission image of a human body; 

Figure 2 is a transmission image corresponding to Figure 1; 

Figure 3 is a functional emission image of a uniform cylindrical phantom; 

Figure 4 is an emission image corresponding to Figure 3 corrected for 
20 scaftering'and attenuation; 

Figure 5 is an x-ray image of a human body showing detailed structural 
information; 

Figure 6 is a block diagram schematically showing the apparatus needed for 
the imaging process; 

25 Figure 7 is a schematic diagram illustrating the apparatus for processing and 

displaying the image data; 

Figure 8 is a flow diagram explaining one embodiment of the invention; 

Figure 9 is a transmission image with the main edges outlined; 

Figure 10 is a mask generated from Figure 9; 
30 Figure 1 1 is an enhanced functional emission image of the human body; 
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Figure 12 shows the image of Figure 1 1 after masking based on a 
transmission image; 

Figure 13 illustrates the display of a functional emission image registered 
with a detailed structural image. 

5 

Figure 6 illustrates schematically the apparatus needed for imaging. A patient 
1 lies on a moveable support 3 beneath a detector 5. The detector is suitable for 
whichever imaging modality is chosen, for instance a gamma or scintillation detector 
for SPECT imaging- The signals from the detector 5 are supplied to an imaging 

1 0 control and processing unit 7 which produces image data for display on display 9. 
For the production of a transmission image a radioactive source 1 1 is provided on the 
opposite side of the patient from the detector 5. Image data may also be acquired 
using multiple projections from different angles of placement of the detector 5 (and 
source 1 1 if necessary). 

1 5 Typically image data is processed by computer before displayed as 

schematically illustrated in Figure 7. Data sets encoding the image are stored in a 
data store 13 and processed by a detector 15 before being displayed on a computer 
display 17. 

Figure 8 illustrates the registration process in accordance with an embodiment 

20 of the present invention. 

Firstly, in step 100 the three different images are obtained, in this example 
one being a functional emission image such as a SPECT image, one a transmission 
image such as a SPTCT image and one a structural image such as an x-ray image. In 
step 104 an enhancement process is carried out on the emission image. The emission 

25 image has a noisy background, and some features are extremely bright in the image: 
the kidneys, the bladder and the liver are very bright as they evacuate the surplus of 
radioactive contrast agent. This makes a histogram of the intensities in the image 
very irregular with a lot of low intensities, a lot of very high intensities and a few 
intermediate intensities. In the enhancement process the intensities are adjusted so as 

30 to attenuate the very bright intensities. Such enhancement may be, for example, a 
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gamma correction, or a histogram equalization, or other process which enhances the 
separation of features in the image. These techniques are standard in computer vision, 
can be found in, for example, "Digital Image Processing", by Nick Efford, Addison 
Wesley, ISBN 0-201-59623-7, which is herein incorporated by reference. 
5 In step 106 the transmission image is segmented to distinguish between 

different areas of the body and background. The aim is to identify areas which are 
not of interest in the image. This is achieved by smoothing the transmission image 
and then detecting the most significant edges in the image. In this example the edge 
detection may be first detecting points of maximum intensity gradient. The intensity 

1 0 at each of the detected points is then collected and plotted in a histogram. A 

threshold value, which is the intensity value most represented in the histogram (the 
mode) is then defined. This intensity value is therefore the modal intensity of those 
pixels which are on a detected edge. Then all of the pixels in the image are examined 
and the image is separated into two regions using the above modal intensity value as 

15 a threshold. The largest connected component or components of these regions are 
then extracted. However, other edge detection methods may be used. A transmission 
image in which the edges have been detected and marked (as a light outline) is shown 
in Figure 9. 

Sometimes data sets may be presented, or stored, which include the emission 
20 image before and after correction for scattering and attenuation (by the transmission 
image), but the transmission image has been discarded. In this situation a version of 
the transmission image can be obtained for U6e in the invention by using the emission 
image before and after correction, because the difference between the before and 
after images and a knowledge of the process of correction allows derivation of the 
25 transmission image used in that process. This derived "transmission image" may 
then be subjected to the edge detection process 106 described above. 

The segmentation process 106 segments the image into, two areas: the body 
without the lungs, and dark areas (lungs and background). In one example the dark 
areas (lungs and background) can then be regarded as a mask as indicated in step 
30 108. A mask generated from Figure 9 is shown in Figure 10. This mask is then 
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applicd to the equalised emission image in step 1 10. The application of the mask to 
the image is relatively easy because of the inherent registration of the transmission 
and emission images. The result of applying such a mask to the enhanced emission 
image of Figure 1 1 is shown in Figure 12. In essence this process corresponds to 

5 setting intensity values to zero (or another constant value) in the emission image of 
the lung and background areas identified in the transmission image. However, it 
should be noted that in fact the segmented transmission image is indicating which 
pixels are inside the body and which are outside. Sometimes it may be useful to use 
pixels outside the body in the registration process, as it can make the method more 

10 robust. But the knowledge provided by the segmented transmission image as to 
which pixels are inside the region of interest and which are outside is still useful 
The registration algorithm can take account of the region not of interest as a class of 
intensities describing a particular area, and in this case they could be identified in the 
mask by setting their intensity value to 1, to include them, rather than 0 to exclude 

15 them. 

The masked emission image is then available for registration with an image 
obtained by another modality, such as an x-ray image. The fact that large areas of the 
image (which are not of interest) have been masked out, or at least identified, makes 
the relevant and useful in&nnaticn in the combined emission and transmission image 

20 more specific. Any of the known matching and registration processes may be used, 
for instance based on detection and comparison of intensities, or intensity 
distributions, or detection and comparison of edges or of geometric structures. For 
example methods based on matching intensities include the calculation of a statistical 
measure based on a joint histogram of the target image and the masked emission 

25 image, and changing the transformation parameters to minimise a similarity criterion. 
Other matching techniques which may be used are described, for example in 
"Accurate Three-Dimensional Registration of CT, PET and/or MR Images of the 
Brain", by Pelizzari, C.A., et al., Journal of Computer Assisted Tomography, Volume 
13, 1989; "MRI-PET Registration with Automated Algorithm ,, by Woods, R.P., et 

30 al., Journal of Computer Assisted Tomography, Volume 1 7, 1993; "The Principle 
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Axis Transformation - A Method for Image Registration", by Albert, N.M., et aL, 
Journal of Nuclear Medicine, Volume 31, 1990; "New Feature Points Based on 
Geometrical Invariance for 3-D Image Registration", Research Report Number 2149 
from the INRIA, Jean-Philippe Thirion; and "A survey of medical Image 

5 Registration", by J.B. Antoine Maintz, M, Viergever, Medical Image Analysis, 2(1): 
1-36, 1998, all of which are herein incorporated by reference. 

Once corresponding areas in the masked emission image and in the other 
modality (e.g. x-ray) image have been identified, a mapping transforation which 
indicates which pixels in the frame of the emission image correspond to which pixels 

10 in the frame of the other modality image is obtained. This allows the two images to 
be displayed superposed on one another in step 1 1 4. Figure 1 3 illustrates such a 
display of two registered images, one a masked emission image and one an x-ray 
image. Comparison of this with Figure 5 shows that an area of high metabolic 
activity (high content of radioactive marker) is positioned in the left-hand region 

1 5 (marked by the two orthogonal lines) clearly visible as a ligjiter area in Figure 1 1 , 
Also, differing levels of radioactive marker are visible in the central region. Because 
Figure 1 1 includes the anatomically detailed X^ray CT scan it is easy for a clinician 
to locate the abnormality (e.g. tumour), for instance to plan surgery, radiotherapy or 
other treatment. 



